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(54) Title: COOPERATIVE NESTING OF MECHANICAL AND ELECTRONIC STABILIZATION FOR AN AIRBORNE CAM- 
ERA SYSTEM 



O (57) Abstract: A method and system for stabilizing images being taken by a video camera using electromechanical stabilization. 
O The stabilization system performs inter-frame stabilization based on the velocity of a vehicle on which the video camera is mounted 
^ and the pen rate of a line-of-sight controller of the video camera, The inter-frame stabilization is performed by a software component 
Q by moving a display area (or viewport) within a larger image area. The stabilization system converts an inter-frame stabilization 
^ adjustment into a pen rate adjustment so that the line-of-sight controller will keep the desired object within the image area of the 
camera. 
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COOPERATIVE NESTING OF MECHANICAL AND ELECTRONIC 
STABILIZATION FOR AN AIRBORNE CAMERA SYSTEM 

TECHNICAL FIELD 

[0001] The described technology stabilizes an image stream created by an 

airborne video camera. 

BACKGROUND 

[0002] If not stabilized, the image streams created by airborne video cameras 

can be practically unusable for human observation because frame-to-frame 
image jitter is excessive. This image jitter typically is caused by small, fast 
pointing errors superposed upon larger-amplitude, slower pointing errors. 

[0003] It is possible to reduce this image jitter to acceptable levels with 

refined mechanical stabilization techniques, stabilizing the line of sight of the 
image so that image jitter amplitude is less than an acceptable limit. Such an 
approach can deliver high-quality image streams from all types of cameras 
(video or film) but leads to large, heavy mechanical systems for support of the 
airborne camera. Such systems are the subject of U.S. Patent Nos. 
5,897,223; 3,638,502; 4,989,466; 4,643,539; and 5,184,521. An approach 
relying purely upon mechanical stabilization leads to heavy and complex 
mechanical systems. Usually, multiple nested mechanical stages of 
stabilization are required, with each stage reducing the image jitter further, 
purely by reduction in jitter of the line of sight 

[0004] It would be desirable to have a technique to reduce jitter and avoid the 

need to have such large, heavy, and expensive mechanical systems. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0005] Figure 1 is a block diagram that illustrates the bases of the 

stabilization system in one embodiment. 
[0006] Figure 2 is a block diagram that illustrates nesting of the two types of 

image stabilization in one embodiment. 
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[0007] Figure 3 is a block diagram that illustrates a coupling from the camera 

pointing system to the stabilization system in one embodiment, 
tooos] Figure 4A is a graph that illustrates a typical error budget for the 

cooperative efforts of the various levels of image stabilization employed by 

the stabilization system in one embodiment 
[0009] Figure 4B is a block diagram illustrating two types of electronic 

stabilization systems. 

[0010] Figure 5 is a diagram illustrating the reference frames used in the 

electromechanical stabilization system. 
[0011] Figure 6 illustrates the pixel offsets on an object from one frame to the 

next in one embodiment. 
[0012] Figure 7 is a flow diagram illustrating the processing of the 

stabilization system in one embodiment. 
[0013] Figure 8 is a flow diagram illustrating the processing of the analyze 

pixel offsets for velocity component in one embodiment. 
[0014] Figure 9 is a flow diagram illustrating the processing of the adjust pixel 

offsets for aircraft velocity in one embodiment. 
[0015] Figure 10 is a flow diagram illustrating the processing of the adjust 

pixel offsets for camera rotation component in one embodiment. 
[0016] Figure 1 1 is a flow diagram illustrating the processing of the adjust for 

image analysis by the electromechanical stabilization in one embodiment. 

DETAILED DESCRIPTION 

[0017] A method and system for stabilizing images being taken by a video 

camera using electromechanical stabilization is provided. In one 
embodiment, the stabilization system performs inter-frame stabilization based 
on the velocity of a vehicle on which the video camera is mounted and the 
pan rate of a line-of-sight controller of the video camera. The inter-frame 
stabilization is performed by a software component by moving a display area 
(or viewport) within a larger image area. The inter-frame stabilization 
removes small-amplitude jitter while accounting for vehicle velocity and 
orientation, pan rate and orientation of the line-of-sight controller, distance to 
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an object within the image, and field of view of the camera. The stabilization 
system converts an inter-frame stabilization adjustment into a pan rate 
adjustment so the line-of-sight controller will keep the desired object within 
the image area of the camera. In this way, the stabilization system uses an 
electronic stabilization to remove small-amplitude jitters and feeds those 
adjustments to a mechanical stabilization to account for large-amplitude jitter. 

[0018] In one embodiment, the stabilization system comprises a video camera 

controlled by a gimbal-based, line-of-sight controller that is mounted on an 
aircraft. While the aircraft is flying, the video camera feeds images to the 
software component that provides the inter-frame stabilization based on the 
scan and tilt rate (i.e., pan rate) of the line-of-sight controller. The software 
component removes small-amplitude jitter while factoring in the scan and tilt 
rate of the line-of-sight controller. The software component receives images 
from the camera that are larger than the display area. The software 
component moves the display area around within the larger image to remove 
the small-amplitude jitter. The software component then calculates a scan 
and tilt rate adjustment for the line-of-sight controller. The software 
component then provides the adjustment to the line-of-site controller so it can 
keep the video camera at the desired line of sight. 

[0019] The stabilization system nests mechanical stabilization and electronic 

stabilization loops to exploit modern capabilities in electronics and image 
processing. Because not all of the stabilization is achieved mechanically, a 
simpler, cheaper, smaller, lighter, and lower-power mechanical gimbal system 
may be used. 

[0020] The stabilization system uses an electronic image stabilization to 

augment mechanical line-of-sight stabilization to achieve full frame-to-frame 
stabilization of the image flow. The mechanical system is used for the large- 
amplitude, slow line-of-sight corrections required, while electronic 
stabilization is used for the small-amplitude, faster corrections not handled by 
the mechanical system. These stabilization loops are nested to take 
advantage of the characteristics of both types of stabilization. The 
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stabilization system can implement various levels of interaction between 
these stabilization methods. 

Inner-to-Outer Nulling 



electronically by "sliding" successive frames in the image stream on the 
display screen or on the focal plane array within the camera. This type of 
stabilization accounts for small amplitudes, typically a small fraction of the 
frame. The stabilization system provides the image correction implemented 
by this electronic stabilization to the mechanical pointing system (i.e., the 
line-of-sight controller) so that the mechanical pointing system can implement 
movements to cause the long-term average electronic image correction to 
tend toward zero. If such corrections are not implemented by the mechanical 
pointing system, then the displayed image might slowly drift and exceed the 
limits of practical correction of the electronic stabilization. 

Outer-to-lnner Coupling 

[0022] A user may want the image to 'flow" across the screen, as for example, 

when the camera is panned while images are being gathered. An electronic 
stabilization system may misinterpret such image flow as unwanted jitter and 
will attempt to correct for it. Such misinterpretation would lead to momentarily 
stabilized images with sudden "steps" required when the electronic correction 
reaches its practical limit. The stabilization system can prevent such sudden 
steps if provided with image flow from the command system of the mechanical 
pointing system. Thus, the stabilization system can be used to enable 
smooth electronic image stabilization, even when the camera is being panned 
across a scene and the image flows across the display screen. 

[0023] Figure 1 is a block diagram that illustrates the bases of the 

stabilization system in one embodiment. Large-amplitude stabilization of 
camera line of sight is implemented with a motor-driven gimbal system 101 
for controlling a camera 105. Small-amplitude stabilization of inter-frame 
motion is accomplished by electronic stabilization techniques implemented 
via an image processor 102. These electronic techniques shift the displayed 



[0021] 



The fastest, smallest-amplitude stabilization is implemented 
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image on a display screen 103 electronically by an adjustment and feed the 
adjustment to a gimbal controller 104. The image processor 102 may include 
a central processing unit, memory, input devices (e.g., keyboard and pointing 
devices), output devices (e.g., display devices), and storage devices (e.g., 
disk drives). The memory and storage devices are computer-readable media 
that may contain instructions that implement the electronic stabilization of the 
video stream provided by the camera 105. In addition, data structures and 
message structures (e.g., gimbal controller command and gimbal scan and tilt 
rate) may be stored or transmitted via a data transmission medium, such as a 
signal on a communications link. 
[0024] Figure 2 is a block diagram that illustrates the adjustment of the pan 

rate of the camera based on the inter-frame adjustment in one embodiment. 
The adjustment to the mechanical pointing system reduces any accumulating 
electronic image re-registration by mechanically re-pointing the camera. This 
allows the stabilization system to stabilize the image stream with a minimal 
loss of image size. 

[0025] Figure 3 is a block diagram that illustrates the adjustment of the inter- 

frame stabilization based on the pan rate of the camera in one embodiment. 
This adjustment enables the stabilization system to stabilize successive 
images not simply in the display frame but rather to stabilize only unwanted 
motion between successive images. Thus, when the camera line of sight is 
intentionally panning the scene, the desired image flow is not suppressed by 
the stabilization system. Rather, the stabilization system suppresses only 
deviations from this desired image flow. 

[0026] Figure 4A is a graph that illustrates a typical error budget for the 

cooperative efforts of the various levels of image stabilization employed by 
the stabilization system in one embodiment. Low-speed, large-amplitude 
stabilization is implemented by the mechanical gimbal system, while high- 
speed, small-amplitude stabilization is implemented by the electronic 
stabilization system. 

[0027] Figure 4B is a block diagram illustrating two types of electronic 

stabilization systems. Gyro-to-image electronic stabilization 401 measures 
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mechanical motion of the camera body and implements image re-registration 
to compensate for this mechanical motion. Such electronic image 
stabilization is implemented in some hand-held "camcorder" video cameras 
sold in retail stores by Sony, Panasonic, and others. Gyro-to-image 
electronic stabilization is most practically implemented in airborne hardware 
and software within an airborne camera system. Frame-to-frame electronic 
image stabilization 402 employs image recognition techniques to re-register 
every frame prior to its display in a succession of image frames. Frame-to- 
frame stabilization attempts to minimize unwanted shift of successive frames 
on the display screen. 

[0028] Figure 5 is a diagram illustrating the reference frames used in the 

stabilization system in one embodiment. The reference frame of the earth is 
represented by north N, east E, and down D coordinates. The position of an 
aircraft, which may be provided by an altimeter and a GPS system, is in the 
earth reference frame. The reference frame of the body of the aircraft is 
represented by heading B1, pitch B2, and roll B3 coordinates that may be 
provided by the aircraft's gyros. The reference frame of the camera is 
represented by a line of sight C1, tilt C2, and scan C3 coordinates. In one 
embodiment, the camera is controlled by an inertia! stabilization system that 
controls the gimbal motors to control the orientation of the C1, C2, and C3 
camera axes. The electromechanical stabilization system receives camera 
scan and tilt rate information from the camera rate gyros and adjusts these 
rates to further account for frame-to-frame jitter information and image 
recognition, which are provided by the electronic stabilization. 

[0029] The stabilization system inputs the images generated by the camera, 

the velocity of the aircraft in the earth reference frame V*^, the camera 
scan rate and tilt rate, the orientations of the aircraft and the camera, and the 
distance to an object in the images. The stabilization system analyzes 
consecutive frames and determines the optimal translation of one frame to 
make it best coincide with the preceding frame. The stabilization system may 
use conventional pattern recognition techniques to locate the object within 
the image. The stabilization system provides an offset in pixels to best 
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superimpose one frame onto the next frame. The pixel offsets may be 
represented as the number of pixels horizontally (also referred to as the scan 
direction) and the number of pixels vertically (also referred to as the tilt 
direction) on the display. In one embodiment, the stabilization system has an 
image buffer in memory that is larger than the displayed image. When the 
stabilization system detects variations in the position of an object resulting 
from jitter, it can offset the displayed image by the calculated scan and tilt 
offset, providing a frame that best superimposes with the previous displayed 
frame, thus effectively removing the jitter. 

[0030] Figure 6 illustrates the pixel offsets on an object from one frame to the 

next in one embodiment. The memory buffer stores the image received from 
the camera using 2000 by 2000 pixels. However, only 1800 by 1800 pixels 
are displayed on the display. Thus, the image can be adjusted by 100 pixels 
in the negative and positive scan and tilt directions to account for jitter, 
assuming the last frame was centered in the memory buffer. In this example, 
a car is at position (1 000, 1 000) in the first frame, and the upper left comer of 
the display corresponds to position (101, 101). In a subsequent frame, the 
car is now at position (1010, 1020). Thus, the stabilization system can 
display position (111, 121) as the upper left corner of the display to place the 
car at the same position on the display from one frame to the next frame. 

[0031] Since the camera may be panning a scene and the aircraft platform 

may be moving relative to the scene, a portion of the pixel offsets calculated 
by the stabilization system may be a result of this desired movement. In such 
a case, the stabilization system is provided with aircraft velocity and 
orientation, camera line of sight and orientation, and camera scan and tilt rate 
to estimate and factor out this desired movement before adjusting the image. 
The stabilization system calculates the sum of the pixel offsets resulting from 
the aircraft velocity and orientation, and the camera orientation and angular 
rate. The stabilization system then subtracts this sum from the pixel offsets 
calculated from the image analysis to give the pixel offsets attributable to the 
jitter. 
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[0032] Because the number of offset pixels is limited, the electromechanical 

stabilization loop keeps the desired image in the center of the camera. To do 
so, the stabilization system uses the pixel offsets to re-center the gimbal 
angles of the camera. The stabilization system converts the pixel offsets to 
corresponding scan and tilt error. The stabilization system also calculates 
scan and tilt rates. It then adjusts the scan and tilt rate of the camera to track 
an object or prevent an overflow of the pixel offset in the stabilization system. 

[0033] The stabilization system uses transformation matrices to represent the 

current orientation of the body of the aircraft relative to the earth reference 
frame and the current orientation of the camera to the body reference frame. 
The camera reference frame relative to the body of the plane reference frame 
is represented by a transformation matrix Ccb for transforming a vector from 
the body reference frame to the camera reference frame. Ccb is a 3-by-3 
matrix whose columns are orthogonal and normalized, also referred to as a 
matrix of direction cosines. The following equation represents the conversion 
of a position in the body reference frame to the camera reference frame: 

R^C^R 3 (1) 

where R B represents the position in the body reference frame and R c 
represents the position in the camera reference frame. An example Ccb is 
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The matrix Ccb is set based on the angles of the gimbal relative to the body. 
Thus, this matrix represents the current gimbal angles. A matrix Cbb is for 
transforming from the earth reference frame to the body reference frame. 
Thus, the matrix Cbe represents the heading, pitch, and roll of the aircraft as 
measured by the gyro of the aircraft. 
[0034] Figure 7 is a flow diagram illustrating the processing of the 

stabilization system in one embodiment. The stabilization system calculates 
an initial image adjustment based on analysis of the image. The stabilization 
system then adjusts the initial adjustment based on velocity and orientation of 
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the aircraft and the scan and tilt rate of the camera. In block 701, the system 
invokes a component to analyze the image and calculate image pixel offsets. 
The invoked component provides the scan and tilt pixel offsets. The 
component may use standard object recognition techniques to recognize an 
object in successive frames and set the initial adjustment based on the 
difference in locations of the object. In block 702, the system invokes the 
adjust pixel offset component, which determines the velocity pixel offsets that 
are attributable to the desired aircraft velocity, the rotation of the camera, and 
the orientations of the aircraft and the camera. In blocks 703-704, the system 
calculates pixel offsets due to unwanted image jitter by subtracting the 
velocity pixel offsets and desired camera rotation pixel offsets from the image 
pixel offsets. In block 705, the stabilization system displays the stabilized 
image by selecting the portion of the screen that begins at the newly 
calculated offsets. In block 706, the system invokes the adjust for image 
analysis component providing the pixel offsets to calculate a change in the 
scan and tilt rate of the camera to help ensure that the image correction will 
not exceed the maximum number of allowable pixels. In block 707, the 
system adjusts the scan and tilt rate of the camera. The system may be 
invoked to process every frame or only a subset of the frames (e.g., every 10 
frames). 

[0035] Figure 8 is a flow diagram illustrating the processing of the analyze 

pixel offsets for velocity component in one embodiment. The component is 
provided with the velocity of the aircraft V?^ in the earth reference frame, 
the orientation of the aircraft Cue, and the orientation of the camera Ccu. In 
block 801, the component invokes the adjust pixel offsets for aircraft velocity 
component to calculate the pixel offsets attributable to the velocity of the 
aircraft. In block 802, the component invokes the adjust pixel offsets for 
camera rotation component to calculate the pixel offsets attributable to the 
rotation of the camera. In blocks 803-804, the component combines the pixel 
offsets for the scan and tilt directions. 

[0036] Figure 9 is a flow diagram illustrating the processing of the adjust pixel 

offsets for aircraft velocity in one embodiment. In block 901, the component 
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calculates a transformation matrix for transforming from the earth reference 
frame to the camera reference frame as follows: 

In block 902, the component calculates the line of sight of the camera in the 
earth reference frame as 

L^cJOAO) 1 (4) 

where L E is the line of sight of the camera in the earth reference frame and 
where the superscript T represents the transpose of the matrix or vector. In 
block 903, the component retrieves the distance or range K to the object at 
the center of the camera. The range may be provided by a range finder or by 
calculating the distance using the altitude of the target. For example, if the 
object is at sea level, then the distance can be calculated based on the 
altitude of the aircraft and the angle of the line of sight. In block 904, the 
component transforms the velocity of the aircraft to the camera reference 
frame as 

v L=Ccb*vL (5) 

In block 905, the component normalizes the velocity of the aircraft as 

where V^La is the normalized velocity of the aircraft in radians per hour. For 
example, if the velocity of the aircraft in the scan direction is 100km/hr and 
the distance to the object is 1km, then the normalized velocity is 100rad/hr, 
which means the aircraft moves in the scan direction 100 times the distance 
to the object in one hour. In block 906, the component calculates the 
difference in scan units as 

AS c ="vl raft (S)*AT (7) 

where AT is the frame refresh period. For example, when the normalized 
velocity is 100rad/hr and the refresh rate is 15 times per second, then the 
change in scan units is: 
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lOOratf Ihr Isec 1 

■ m l^r^-TT = T7K rad=Ude Sr ees (8) 



hr 3600sec 15 540 

In block 907, the component calculates the aircraft pixel offset in the scan 
direction by converting the difference in scan units to the corresponding pixel 
offset factoring in the field of view (or zoom) of the camera. The component 
calculates the pixel offset as 

APO(S)=AS c *P/Z (9) 

where APO(S) is the pixel offset in the scan direction, Z is the zoom factor, 
and P is the pixel density. For example, if the scan units are 1/540rad and 
there are 2000 pixels in the scan direction with a field of view of .93rad (1km 
field of view at 1 km distance), the pixel offset is 

\rad 2000p/xe& „ . , 

In blocks 908-909, the component calculates the pixel offset in the tilt 
direction in a similar manner. 
[0037] Figure 10 is a flow diagram illustrating the processing of the adjust 

pixel offsets for the camera rotation component in one embodiment. The 
component receives the instantaneous camera scan and tilt rates from the 
gimbal controller. Alternatively, the component can calculate them based on 
the orientations of the aircraft and the camera. In block 1001, the component 
calculates the difference in scan units as 

AS C =IS*AT (11) 

where IS is the instantaneous scan rate of the camera measured by a rate 
gyro. In block 1002, the component calculates the difference in tilt units as 

AT C = IT* AT (12) 

where IT is the instantaneous tilt rate of the camera measured by a rate gyro. 
In block 1003, the component calculates the camera pixel offset in the scan 
direction by converting the difference in scan units to the corresponding pixel 
offset, factoring in the field of view (or zoom) of the camera. In block 1004, 
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the component calculates the pixel offset in the tilt direction in a similar 
manner. 

[0038] Figure 1 1 is a flow diagram illustrating the processing of the adjust for 

image analysis by the stabilization in one embodiment. The component is 
passed the pixel offsets corresponding to the adjustments made by the 
stabilization system. In blocks 1101-1102, the component converts the pixel 
offsets to corresponding scan angle in radians and tilt angle in radians, 
factoring in the field of view of the camera. In blocks 1103-1104, the 
component calculates the angle error resulting from the pixel offsets as 

AE(S) = AS C *K (13) 
AE(T) = AT C *K (14) 

where AE is the angle error and K is the distance to the center of the image. 
In blocks 1 105-1 1 06, the component calculates the adjustments for the angle 
errors as 

A(S) = (W/AT)*AE(S) (15) 
A(T) = (W/AT)*AE(T) (16) 

where A(S) is the adjustment for the scan rate in radians per second and W is 
a weighting factor that controls the bandwidth of the feedback loop. The 
weighting factor controls the speed at which adjustments can be made to the 
scan and tilt rates. The stabilization system compares the adjustment to the 
scan rate of the camera provided by the gyro and uses the difference in rate 
to control the velocity of the gimbal motors. 
[0039] All patents and articles cited herein are hereby incorporated by 

reference in their entirety and relied upon. Further details of methods of 
operating airborne cameras in accordance with other embodiments of the 
invention are described in U.S. Patent Application No. 60/440,983, filed on 
January 17, 2003, and entitled, "Compensation for Overflight Velocity When 
Stabilizing an Airborne Camera," and U.S. Patent Application No. 
60/440,977, filed on January 17, 2003, and entitled, "Method and Apparatus 
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for Stabilizing Payload, Including Airborne Cameras," which are hereby 
incorporated by reference. 
[0040] One skilled in the art will appreciate that although specific 

embodiments of the stabilization system have been described herein for 
purposes of illustration, various modifications may be made without deviating 
from the spirit and scope of the invention. For example, the principles of the 
stabilization system may be used on a transport mechanism other than an 
airplane, such as a satellite, a rocket, a missile, a train, an automobile, and 
so on. In addition, the camera may even be stationary or not traveling 
relative to an object in the video. Accordingly, the invention is not limited 
except by the appended claims. 
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CLAIMS 



[d] 1 . A method for stabilizing an image of an object being taken from 

a video camera, the video camera being moved by a transport mechanism 
and being controlled by a line-of-sight controller, the method comprising: 
receiving a plurality of images of the object; and 
for each of the plurality of received images, 

receiving a velocity and orientation of the transport mechanism; 
receiving an orientation of the camera relative to the transport 

mechanism; 
receiving a scan and tilt rate of the camera; 
receiving a distance from the camera to the object; 
calculating an inter-frame stabilization adjustment based on the 
velocity and orientation of the transport mechanism, the 
orientation of the camera, the scan and tilt rate of the 
camera, and distance to the object; 
calculating a line-of-sight adjustment for the line-of-sight 

controller based on the inter-frame stabilization; and 
controlling the line-of-sight controller in accordance with the 
calculated line-of-sight adjustment. 

[c2] 2. The method of claim 1 wherein the transport mechanism is an 

airborne vehicle. 

[c3] 3. The method of claim 1 wherein the line of sight of the camera is 

derived from the line-of-sight controller. 

[c4] 4. The method of claim 1 including adjusting the display of the 

images based on the inter-frame stabilization adjustment. 
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[c5] 5. The method of claim 4 wherein the received images are larger 

than the displayed images and the adjusting of the display of the images 
moves an area of a displayed image within the received image. 

[c6] 6. The method of claim 4 wherein the inter-frame stabilization 

adjustment specifies the number of pixels in scan and tilt directions. 

[c7] 7. The method of claim 1 wherein the controlling of the line-of- 

sight controller specifies rate of scan and tilt movement 

Ic8] 8. The method of claim 1 wherein the distance to the object is 

provided by a range finder. 

[c9j 9. The method of claim 1 wherein the distance to the object is 

calculated based on the line of sight of the camera and the difference in 
altitude of the object and the camera. 

[do] 10. The method of claim 1 wherein the velocity of the transport 

mechanism is relative to the object. 

[di] 11. The method of claim 1 wherein the velocity of the transport 

mechanism is relative to an earth frame of reference. 

[d2] 12. The method of claim 1 wherein the calculated inter-frame 

stabilization adjustment factors in field of view of the display. 

[d3] 13. A method for stabilizing an image of an object being taken from 

a video camera, the video camera being moved by a transport mechanism 
and being controlled by a line-of-sight controller, the method comprising: 

determining a difference in the location of the object within the image 
from one frame to the next frame; 
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calculating an inter-frame stabilization adjustment based on the 

determined difference; 
calculating a line-of-sight adjustment for the line-of-sight controller 

based on the inter-frame stabilization; and 
controlling the line-of-sight controller in accordance with the calculated 

line-of-sight adjustment. 

[d4] 14. The method of claim 13 wherein the determining of the 

difference includes analyzing scan and tilt rate of the line-of-sight controller. 

[d5] 15. The method of claim 13 wherein the determining of the 

difference includes analyzing velocity of the transport mechanism. 

[ci6] 16. The method of claim 13 wherein the determining of the 

difference includes analyzing line of sight of the camera. 

[d7] 17. The method of claim 13 wherein the determining of the 

difference includes analyzing orientation of the camera and the transport 
mechanism. 

[d8] 18. The method of claim 13 wherein the determining of the 

difference includes recognizing the object within the images. 

[ci9] 19. The method of claim 13 wherein the calculated line-of-sight 

adjustment specifies a scan and tilt rate for the line-of-sight controller. 

[c20] 20. An apparatus for stabilizing imagery from an airborne video 

camera, comprising: 

a mechanical line-of-sight controller for controlling line of sight of the 

video camera at a specified line-of-sight adjustment rate; and 
an electronic stabilization component that provides frame-to-frame 
image stabilization based on the specified line-of-sight 
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adjustment rate and that provides to the mechanical line-of-sight 
controller a new line-of-sight adjustment rate derived from an 
amount of frame-to-frame image stabilization. 

[c2i] 21. The apparatus of claim 20 wherein the amount of frame-to- 

frame image stabilization is additionally based on velocity and orientation of 
an airborne transport vehicle, orientation of the camera relative to the 
airborne transport vehicle, and distance from the camera to an object within 
the image. 

[c22] 22. The apparatus of claim 20 wherein the line-of-sight adjustment 

rate includes a scan rate and a tilt rate. 

[c23] 23. The apparatus of claim 20 wherein an image received from the 

video camera is larger than a displayed image and the electronic stabilization 
component provides frame-to-frame image stabilization by adjusting the 
location of the displayed image within a received image. 

[c24] 24. The apparatus of claim 20 wherein the specified line-of-sight 

adjustment rate includes a user-specified image flow. 

[c25] 25. The apparatus of claim 20 wherein the mechanical line-of-sight 

controller is a motorized gimbal system. 

[c26] 26. The apparatus of claim 20 wherein the frame-to-frame 

adjustment keeps an object of the images at the same location when 
displayed. 

[c27] 27. An apparatus for stabilizing imagery from an airborne video 

camera, comprising: 

a mechanical line-of-sight controller for controlling line of sight of the 
video camera at a specified line-of-sight adjustment rate; and 
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an electronic stabilization component that provides frame-to-frame 
image stabilization based on location of an object within the 
images and that provides to the mechanical line-of-sight 
controller a new line-of-sight adjustment rate derived from an 
amount of frame-to-frame image stabilization. 

I 028 ! 28. The apparatus of claim 27 wherein an amount of frame-to-frame 

image stabilization is additionally based on velocity and orientation of an 
airborne transport vehicle, orientation of the camera relative to the airborne 
transport vehicle, and distance from the camera to the object within the 
image. 

[o29] 29. The apparatus of claim 27 wherein an amount of frame-to-frame 

image stabilization is additional based on the specified line-of-sight 
adjustment rate. 

[c30] 30. The apparatus of claim 27 wherein the line-of-sight adjustment 

rate includes a scan rate and a tilt rate. 

I 031 ! 31 . The apparatus of claim 27 wherein an image received from the 

video camera is larger than a displayed image and the electronic stabilization 
component provides frame-to-frame image stabilization by adjusting the 
location of the displayed image within a received image. 

[c32] 32. The apparatus of claim 27 wherein the specified line-of-sight 

adjustment rate includes a user-specified image flow. 

[c33] 33. The apparatus of claim 27 wherein the mechanical line-of-sight 

controller is a motorized gimbal system. 

[c34j 34. The apparatus of claim 27 wherein the frame-to-frame adjust- 

ment keeps the object of the images at the same location when displayed. 
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[c35] 35. A method for stabilizing images being taken from a video 

camera mounted on a moving vehicle, the camera having a line of sight being 
controlled by a line-of-sight controller, the method comprising: 

calculating inter-frame stabilization adjustments to account for velocity 
of the vehicle; 

displaying the images in accordance with the calculated inter-frame 

stabilization adjustments; 
calculating line-of-sight adjustments for the line-of-sight controller 

based on the inter-frame stabilization adjustments; and 
controlling the line-of-sight controller in accordance with the calculated 

line-of-sight adjustments. 

[c36] 36. The method of claim 35 wherein the calculating of the inter- 

frame stabilization adjustments factors in scan and tilt rate of the line-of-sight 
controller. 

[c37] 37. The method of claim 35 wherein the calculating of the inter- 

frame stabilization adjustments factors in line of sight of the camera. 

[c38] 38. The method of claim 35 wherein the calculating of the inter- 

frame stabilization adjustments factors in orientation of the camera and the 
vehicle. 

[c39] 39. The method of claim 35 wherein the calculating of the inter- 

frame stabilization adjustments includes recognizing an object within the 
images. 

MO] 40. The method of claim 35 wherein the calculated line-of-sight 

adjustment specifies a scan and tilt rate for the line-of-sight controller. 
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